<html>
<head><meta charset="utf-8"><title>parser/lexer structure · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html">parser/lexer structure</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="196782145"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196782145" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Russell Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196782145">(May 07 2020 at 14:55)</a>:</h4>
<p>I was working on a lexer PR and realized that the parser uses the lexer API in a fairly strange way -- the lexer maintains a cursor structure that't intended to read through the entire input, but no usages ever read more than one token out of it. (They reinitialize it every time)</p>
<p>I assume it's a relic of a different refactoring. I suspect it could even be a mild performance issue since we _reallocate_ the <code>Cursor</code> for every single token. </p>
<p>Is this on purpose? Should this be cleaned up?</p>



<a name="196782979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196782979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196782979">(May 07 2020 at 15:01)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="133169">@matklad</span></p>



<a name="196783245"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783245" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783245">(May 07 2020 at 15:03)</a>:</h4>
<p>Are you talking about this Cursor: <a href="https://github.com/rust-lang/rust/blob/4802f097c86452cd2e09d44e88dbcb8e08266552/src/librustc_lexer/src/cursor.rs#L7-L12" title="https://github.com/rust-lang/rust/blob/4802f097c86452cd2e09d44e88dbcb8e08266552/src/librustc_lexer/src/cursor.rs#L7-L12">https://github.com/rust-lang/rust/blob/4802f097c86452cd2e09d44e88dbcb8e08266552/src/librustc_lexer/src/cursor.rs#L7-L12</a> ?</p>



<a name="196783458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Russell Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783458">(May 07 2020 at 15:04)</a>:</h4>
<p>Yeah -- that cursor, and this usage: <a href="https://github.com/rust-lang/rust/blob/master/src/librustc_parse/lexer/mod.rs#L121" title="https://github.com/rust-lang/rust/blob/master/src/librustc_parse/lexer/mod.rs#L121">https://github.com/rust-lang/rust/blob/master/src/librustc_parse/lexer/mod.rs#L121</a></p>



<a name="196783556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783556">(May 07 2020 at 15:04)</a>:</h4>
<p>It is intended to work only for a single token. The benefit of the current interface is that it's easily restartable from any point</p>



<a name="196783630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783630">(May 07 2020 at 15:05)</a>:</h4>
<p>(as opposed to an interface, which just gives you a stateful iterator of all tokens in the input)</p>



<a name="196783656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Russell Cohen <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783656">(May 07 2020 at 15:05)</a>:</h4>
<p><span aria-label="+1" class="emoji emoji-1f44d" role="img" title="+1">:+1:</span> makes sense</p>



<a name="196783724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783724">(May 07 2020 at 15:05)</a>:</h4>
<p>Mostly, <code>Cursor</code> is just an impl detail of <code>rustc_lexer</code>, the real interface is just "give me the first token for this input".</p>



<a name="196783886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/parser/lexer%20structure/near/196783886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/parser.2Flexer.20structure.html#196783886">(May 07 2020 at 15:06)</a>:</h4>
<p>This interface is meaningfully <em>more</em> restricted than "give me an interator of tokens", becuase you can't, for example, count parenthesis in lexer, and that is a good thing, if, for exapmle, you want to incrementally re-lex a substring</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>